1*692844 


Ttchaical  Rasaorcli  Note  20r 


CHANGE  DETECTION  IN  AERIAL  PHOTO  COVERAGE 
AS  INFLUENCED  BY  METHODS  OF  COMPARISON 


C.  L.  Klingberp  and  C.  L.  Flworth, 
The  Boeing  Company 

and 

S.  Fpstein 

System  Development  Corporation 


Reproduced  by  the 

CLEARINGHOUSE 
•or  feci-. ral  Sciorh^c  &  Tcclui'cal 
1 1  it  01  nation  bpringhcid  Va  ??151 


U.  S.  Army 

Behavioral  Science  Research  Laboratory 
January  1969 


This  document  has  ti»*en  .lpptpvm 


'd  tor  put* l it"  m  toasts  and  sale. 


IS  distribution  IS  unlinnled. 


BEHAVIORAL  SCIENCE  RESEARCH  LABORATORY 
An  activity  of  the  Chief,  Research  and  Development 


J.  E.  UHlKnER 
Director 


JAMES  E.  WIRRIO; 
COL,  GS 
Commanding 


l 


Research  accomplished 

under  contract  to  the  Department  of  Army 

The  Boeing  Company 

Advanced  Systems  Division,  System  Development  Corporation 


NOTICES 


DISTRIBL ,  ION  Primary  distribution  of  this  report  has  been  made  by  BESRL.  Please  address 
correspondence  concerning  distribution  of  leports  to:  U.  S.  Army  Behavioral  Science  Research 
Latmratory,  Attn:  CHDtfSRL.  Room  239.  Commonwealth  Building, 1320  Wilson  Blvd.,  Arlington, 
Virginia  22209. 


FINAL  DISPOSITION  This  report  may  be  destroyed  when  it  is  no  longer  needed.  Please  do  not 
return  it  to  the  U.  S.  Army  Behavioral  Science  Research  Laboratory. 


NOTE:  The  findings  in  this  report  are  not  to  be  construed  as  an  official  Department  of  the 
Army  position,  unless  so  designated  by  other  authorized  documents. 


Technical  Rtstarcli  Not*  20S 


r 


AD 


CHANGE  DETECTION  IN  AERIAL  PHOTO  COVERAGE 
AS  INFLUENCED  DY  METHODS  OF  COMPARISON 


C.  L.  Klingberp  and  C.  L.  El  worth. 
The  Boeing  Company 

and 

S.  Epstein 

System  Development  Corporation 


Abrnham  H.  Birnbaum,  Task  Leader 


SUPPORT  SYSTEMS  RESEARCH  DIVISION 
Joseph  Zeidner,  Chief 


U.  S.  ARMY  BEHAVIORAL  SCIENCE  RESEARCH  LABORATORY 

Office,  Chief  of  Research  and  Development 
Department  of  the  Army 

Room  239,  The  Commonwealth  Building 
1320  Wil-on  Boulevard,  Arlington,  Virginia  22209 


January  1969 


Army  Project  Number 
2Q662704A721 


Contract  Nos.  DA-49-092-ARO-92  and  DA-49-092-ARO-65 

Interpreter  Techniques  a-51 


This  document  has  been  approved  tor  public  release  and  sale;  its  distribution  is  unlimited. 


BESRL  Technical  Research  Reports  and  Technical  Research  Notes  are  intended  for 
sponsors  of  R8.D  tasks  and  other  research  and  military  agencies.  Any  findings 
ready  for  implementation  at  the  time  of  publication  are  presented  in  the  latter  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recommenda¬ 
tions  for  official  action  normally  are  conveyed  to  appropriate  military  agencies 
by  briefing  or  Disposition  Form. 


FOREWORD 


The  Surveillance  Systems  research  program  of  the  U.  S.  Army  Behavioral  Science 
Research  Laboratory  has  as  its  objective  the  production  of  scientific  data  bearing  on  the 
extraction  of  information  from  surveillance  displays  and  the  efficient  storage,  retrieval, 
and  transmission  of  this  information  within  an  advanced  computerized  image  interpreta¬ 
tion  facility.  Research  results  are  used  in  future  systems  design  and  in  the  development 
of  enhanced  techniques  for  all  phases  of  the  interpretation  process.  Research  is  con¬ 
ducted  under  Army  RDT&E  Project  2Q662704A721 ,  "Surveillance  Systems:  Ground  Sur¬ 
veillance  and  Target  Acquisition  Interpreter  Techniques,"  FY  1969  Work  Program. 

BESRL  research  in  this  area  is  conducted  as  an  in-house  research  effort  augmented 
by  research  contracts  with  organizations  selected  as  having  unique  capabilities  and 
facilities  for  research  in  aerial  surveillance.  The  present  study  was  conducted  jointly  by 
personnel  of  the  Boeing  Company  and  the  System  Development  Corporation  and  of  the 
Behavioral  Science  Research  Laboratory  under  program  direction  of  A.  H.  Bimbaum. 

The  INTERPRETER  TECHNIQUES  Work  Unit  seeks  to  develop  methods  and  procedures 
which  maximize  the  accuracy,  completeness,  and  speed  with  which  intelligence  informa¬ 
tion  is  derived  from  imagery,  both  conventional  aerial  photographs  and  the  products  of 
advanced  sensor  techniques.  The  present  publication  reports  on  a  study  of  methods  of 
comparing  current  with  prior  imagery  cover  of  a  given  area  as  means  of  detecting  change 
in  the  status  of  targets  in  the  area. 


CHANGE  DETECTION  IN  AERIAL  PHOTO  COVERAGE  AS  INFLUENCED  BY  METHODS  OF 
COMPARISON 


BRIEF 


Requirement: 

To  investigate  the  effects  of  selected  variations  in  procedure-methods  of  comparing  current  with 
prioi  imagery  cover,  familiarity  of  the  interpreter  with  the  area,  and  time  allowed  for  interpretation-on 
the  completeness  and  accuracy  with  which  change  in  the  status  of  targets  is  detected. 


Procedure: 

Sixty-four  student  interpreters  performed  a  target  (militory  vehicles)  change  detection  task.  Compar¬ 
ison  materials  provided  were  either  pairs  of  unannotated  photos,  pairs  of  photos  o  e  of  which  was  anno¬ 
tated,  or  schematic  photo  overlays  plus  written  reports  of  earlier  coverage,  paired  with  a  more  recent 
photo.  In  a  fourth  procedure,  separate  analyses  of  the  comparative  photos  were  made  and  ihe  data  sub¬ 
jected  to  a  simulated  computer  comparison  procedure.  Each  comparative  scene  was  viewed  for  either  5  or 
15  minutes.  Half  the  subjects  had  gained  familiarity  with  the  target  areas  through  a  prior  interpretation 
and  half  had  not.  Change  detection  performance  was  evaluated  in  terms  of  completeness  and  accuracy. 


Findings: 

Under  the  5-minute  time  limit,  the  computer  comparison  of  independent  interpretations  nf  current  and 
prior  photo  coverage  resulted  in  more  complete  change  detection  than  did  the  other  methods  employed  and 
about  the  same  degree  of  accuracy.  The  advantage  of  computer  comparison  was  less  marked  under  the 
15-minute  time  limit. 

Providing  the  interpreter  with  overlay  plus  a  written  report  on  the  prior  imagery  was  the  least  effec¬ 
tive  of  the  comparison  methods  investigated.  Without  the  actual  imagery  from  the  earlier  mission,  the 
interpreter  has  no  wjy  of  correcting  errors  in  the  earlier  interpretation.  The  method  also  was  time-con¬ 
suming 

Interpreter  familiarity  with  the  terrain  through  preparation  of  reports  on  the  prior  imagery  resulted  in 
more  complete  end  more  accurate  change  detection  under  the  5-minute  time  limit.  Under  the  15-minute 
tune  lur  the  advantage  was  negligible. 


Utilisation  of  Findings 

When  fast  change  detection  is  necessary  and  work  time  allotted  is  short,  more  complete  change 
detection  can  be  attained  by  use  of  a  computerized  system  of  comparing  data  extracted  separately  from 
current  and  prior  photographic  imagery,  Consideration  of  such  a  procedure  has  been  recommended  for 
advanced  surveillance  systems. 

When  the  comparison  for  change  detection  is  to  be  made  by  an  interpreter,  there  is  an  advantage, 
under  extreme  time  pressure,  in  having  the  interpreter  of  the  prior  imagery  make  the  comparison. 

Inspection  of  current  imagery  and  use  of  overlays  and  reports  based  on  prior  imagery  should  be  used 
to  obtain  information  on  target  change  status  only  if  none  of  the  alternative  methods  described  is  feasible. 
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CHANGE  DETECTION  IN  AERIAL  PHOTO  COVERAGE  AS  INFLUENCED  BY  METHODS  OF 
COMPARISON 


THE  PROBLEM 


Need  for  Multiple  Cover  Imagery 

The  Increased  mobility  of  military  personnel ,  weapons,  systems,  and 
support  equipment,  and  the  greater  firepower  capability  of  small. opera¬ 
tional  units  have  placed  new  emphasis  on  need  for  comprehensive  and  timely 
military  intelligence  information.  Crucial  to  maintaining  full  cognizance 
of  enemy  activity  is  the  detection  of  changes  in  the  status  and  deployment 
of  military  forces  through  acquisition  and  interpretation  of  comparative- 
cover  aerial  photography.  Detailed  inspection  of  aerial  photographs  taken 
of  the  same  terrain  at  different  times  can  provide  critical  intelligence 
information  regarding  the  enemy's  strength,  deployment,  activity,  and 
battle  plan  not  readily  available  from  other  sources. 

Comparative-cover  change  detection  performance  is  affected  by  many 
situational  factors.  The  training  and  experience  of  the  interpreter  (both 
general  and  specific  to  the  task  and  terrain),  the  interpretation  proce¬ 
dures  followed,  the  amount  and  nature  of  the  comparative  materials  the 
interpreter  has  available,  the  amount  and  characteristics  of  the  display 
and  support  equipment  he  uses,  and  the  stated  or  inferred  requirements  of 
the  intelligence  task,  all  contribute  to  the  accuracy,  completeness,  rele¬ 
vance,  and  timeliness  of  the  interpreter's  report. 

Among  the  numerous  factors  influencing  change  detection  performance, 
of  immediate  significance  are  1)  type  and  amount  of  information  on  the 
earlier  coverage  and  how  the  earlier  coverage  is  used,  2)  the  interpreter's 
familiarity  with  the  target  area,  3)  time  allocated  for  the  change- 
detection  task,  4)  similarity  of  photo  scale  and  directional  orientation 
of  the  early  and  late  coverage,  and  5)  compatibility  of  photo  dissimilar¬ 
ities  with  display  device  requirements.  The  present  study  was  concerned 
primarily  with  the  first  three  of  these  procedural  variables.  The  objec¬ 
tive  was  to  study  the  effect  of  variations  in  these  procedures  on  the 
completeness  and  accuracy  of  change  detection  from  tactical  comparative 
cover  imagery. 


Variations  in  Methods  of  Comparison 

From  among  the  numerous  forms  of  comparison  materials  an  interpreter 
might  use  in  detecting  changes  in  a  given  section  of  terrain,  four  were 
selected  as  representative  of  the  range  of  situations  encountered  under 
most  operational  conditions: 


Unannotated  Photos .  Perhaps  the  most  frequent  procedure  is  a 
straightforward  comparison  of  two  unannotated  photos  (UP)  taken  of  a 
target  area  at  different  times.  The  interpreter  searches  each  photo, 
scanning  back  and  forth  between  the  two  to  find  the  targets  and  deter* 
mine  their  change  status. 

Annotate  '  Photos.  Often  the  earlier  photo  (  P^ )  has  to  be  retrieved 

from  a  storage  file  and  combined  with  a  newly  acquired  photo  (P?)  to  form 

a  comparative-cover  pi  ito  pair.  The  earlier  photo  may  have  been  analyzed 
when  it  was  first  obtained,  and  annotated  in  the  process  to  indicate  the 
targets  detected.  This  annotated  photograph  (AP),  may  have  been  annotated 
by  the  same  interpreter  who  is  making  the  comparison,  or  the  annotations 
may  have  been  made  by  some  other  interpreter. 

Overlay  plus  Written  Report ■  In  addition  to  annotating  photographs 
in  the  process  of  interpreting  them,  it  is  almost  always  a  requirement 
that  a  written  report  of  the  findings  also  be  prepared.  Content  and 
format  of  these  reports  will  vary  as  dictated  by  objectives  of  the  mission 
for  which  they  were  prepared.  Schematic  situation  overlays  or  photo  over¬ 
lays  may  also  have  been  prepared,  and  may  be  found  with  the  written  re¬ 
port  in  the  files.  The  combined  situation  overlay  and  written  report  (OR) 
prepared  from  the  earlier  imagery  serves  as  comparison  material  for  assess¬ 
ment  of  target  changes  in  the  more  recent  imagery. 

Computer  Comparison,  in  the  computerized  comparison  condition  (CC), 
the  interpreter  analyzes  the  late  cover  only.  The  computer  makes  the 
earlier  image  sample  comparison  and  prepares  the  change  detection  report. 
With  the  advent  of  small  field  model  computerized  information  storage  and 
retrieval  systems,  it  may  become  possible  for  previously  acquired  intelli¬ 
gence  data  on  the  location  and  identification  of  enemy  forces  to  be  encoded 
and  stored  in  the  computer's  memory  bank  for  future  query  and  recall.  The 
interpreter  would  have  only  to  locate  and  identify  targets  on  the  new 
photographs  and  then  query  the  computer  as  to  the  presence  or  absence  of 
any  previously  reported  target  at  a  given  location  ( spe  .ified  by  means  of 
a  compatible  coordinate  system).  The  interpreter  could  then  make  state¬ 
ments  about  targets  common  to  early  and  later  coverage  and  also  identify 
targets  found  in  the  new  Imagery  but  not  in  the  earlier  imagery.  By  re¬ 
questing  a  complete  readout  on  the  location  and  identification  of  all 
targets  previously  reported,  he  can  determine  which  targets  are  gone  from 
the  area  or  are  new  to  the  area.  The  more  sophisticated  computer  may  even 
perform  the  change  detection  comparison  internally,  requiring  only  the 
encoded  data  on  target  location  and  identification  from  the  two  image 
samples.  The  computer  then  compares,  integrates,  itemizes,  and  prints 
out  information  on  location  and  identification  of  targets  and  change 
status.  The  accuracy  and  completeness  of  such  a  computer-generated  re¬ 
port  is  dependent,  however,  on  the  accuracy  and  completeness  of  the  two 
independent  Inputs  which  are  to  be  compared.  It  is  this  sophisticated 
computer  system  that  was  simulated  for  experimentation  in  the  present 
study. 
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Area  Specialization  of  Interpreter 


A  second  variable  selected  for  investigation  was  whether  a  given 
interpreter  should  be  assigned  prime  responsibility  for  specific  terrain 
areas  and  all  imagery  of  those  areas  routed  to  him.  The  hypothesis 
being  tested  was  that  familiarity  with  a  specific  locale  provides  the 
interpreter  with  an  accumulated  background  of  critical  information  about 
the  area  such  that  he  can  more  accurately  and  rapidly  perform  subsequent 
comparative-cover  analyses  of  the  area.  In  the  present  study,  an  inter¬ 
preter  who,  prior  to  engaging  in  the  change  detection  task,  had  inter¬ 
preted  earlier  coverage  of  a  given  area  was  considered  an  "experienced" 
interpreter,  while  one  who  was  looking  at  a  given  area  for  the  first 
time  was  considered  to  have  "no  experience".  With  the  unannotated  photo 
experimental  condition,  the  experienced  operator  had  interpreted  the 
earlier  imagery  at  some  time  prior  to  the  change  detection  task,  whereas 
the  no-experience  interpreter  had  had  no  previous  experience  with  either 
of  the  comparative  cover  pair.  In  the  case  of  the  annotated  photos  and 
the  overlay  plus  written  report  conditions,  the  annotations  and  other 
reference  materials  had  been  prepared  by  the  experienced  interpreter, 
while  the  no-experience  interpreter  had  to  rely  on  reference  materials 
generated  by  some  other  interpreter.  For  experienced  interpreters  per¬ 
forming  under  the  simulated  computer-comparison  condition,  the  data  on 
the  earlier  imagery  (  )  against  which  the  P^  input  was  compared  were 

provided  by  the  same  subject  as  the  P  data;  P0  information  provided  by 

d  d 

no-experience  interpreters  was  compared  with  P^  information  supplied  by 
some  other  interpreter. 


Time  Requirements 

With  the  increased  perishability  of  reconnaissance  information  in 
today's  highly  mobile  warfare  tactics,  the  rate  at  which  intelligence 
data  can  be  extracted  from  imagery  becomes  of  paramount  importance.  Two 
work  periods,  one  judged  to  be  sufficiently  short  (5  minutes)  to  permit 
assessment  of  information  loss  through  the  effects  of  time  pressure,  and 
the  other  (15  minutes)  long  enough  to  permit  most  subjects  to  complete 
the  task  easily,  were  established  and  imposed  on  the  subjects. 


METHOD 


Experimental  Design 

The  four  methods  of  comparing  early  and  late  imagery,  the  two  levels 
of  inte vpreter  specialization  (experience  with  the  target  area  vs  no  ex¬ 
perience),  and  two  time  limits  (5  minutes  and  15  minutes)  were  combined 
into  the  16  experimental  conditions  shown  schematically  in  Figure  1. 

Each  of  16  matched  groups  of  four  subjects  each  performed  the  target 
search,  target  identification,  and  change  detection  task  under  one  ex¬ 
perimental  condition  (Table  1).  Measures  of  completeness  and  accuracy 
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of  change  detection  without  requirement  of  target  Identification  and  like 
maaaurea  with  requirement  of  target  identification  by  category  were  ob¬ 
tained.  Analysis  of  variance  was  conducted  separately  for  each  of  the 
two  levels  of  identification. 


Subjects 

Subjects  were  64  student  interpreters  (enlisted  men)  nearing  comple¬ 
tion  (12th  week  of  the  15-week  course)  of  the  image  interpreter  course 
at  the  U.  S.  Army  Intelligence  School ,  Fort  Holabird,  Maryland.  Partici¬ 
pants  were  drawn  from  several  classes.  Sixteen  groups  of  four  each  were 
formed,  groups  being  matched  on  scores  on  the  General  Technical  Aptitude 
Area  (GT),  a  composite  of  the  Verbal  and  Arithmetic  Reasoning  tests  of 
the  Army  Classification  Battery  (See  Table  A-l  of  Appendix  A). 


COMPARISON 

METHOD 


Figure  1.  Schematic  representation  of  experimental  conditions 
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Table  1 


SUMMARY  OF  EXPERIMENTAL  CONDITIONS 


Group 

No. 


Comparison  Materials* 


P2  only 
P2  only 

Pg  and  P^  unannotated 

Pg  and  P^  unannotated 

Pg  and  P^  annotated 

Pg  and  P^  annotated 

Pg  and  overlay  and 
Report  Form  of  P^ 

Pg  and  overlay  and 
Report  Form  of  P^ 

Pg  only 

Pg  only 

Pg  and  P^  unannotated 

Pg  and  P^  unannotated 

Pg  and  P^  annotated 

Pg  and  p^  annotated 

Pg  and  overlay  and 
Report  Form  of  P^ 

P0  and  overlay  and 
Report  Form  of  P, 


Method  of 
Comparison 


Experience  Time 
with  Area  Limit 


No  Experience  5 
No  Experience  15 
No  Experience  5 
No  Experience  15 
No  Experience  5 
No  Experience  15 
No  Experience  5 


No  Experience  15 


Experienced 

Experienced 

Experienced 

Experienced 

Experienced 

Experienced 

Experienced 


Experienced  15 


pt  -  earlier  photo;  p2  later  photo 


The  Experimental  Task 

The  task  to  be  performed  was  the  preparation  of  updated  annotated 
photographs  and  written  reports  for  ten  different  scenes  in  which  tacti¬ 
cal  military  targets  appeared.  The  process  of  updating  included  the 
following  activities:  target  detection  and  localisation,  target  identi¬ 
fication,  change  detection,  photo  annotation,  and  preparation  of  report. 

Change  Detection  without  Target  Identification.  This  level  of 
identification  required  only  that  the  interpreter  state  whether  an 
object  was  a  target  or  a  non-target.  Having  determined  a  detected 
object  to  be  a  target,  the  Interpreter  then  checked  the  comparative 
photo  to  determine  whether  the  target  was  present  or  absent  at  the  same 
location.  If  he  found  the  target  present  only  on  the  earlier  imagery, 
he  labeled  it  "Gone"  on  the  second  photo;  if  he  found  the  reverse,  he 
labeled  the  target  "New"  on  the  second  photo.  If  the  object  was  seen 
as  present  in  the  same  location  on  both  photos,  the  interpreter  was  re¬ 
quired  to  make  a  same-different  judgment  prior  to  assigning  a  change 
status  of  "Unchanged"  or  "Replaced".  Erroneous  change  status  assign¬ 
ments  could  thus  result  from  one  or  more  of  several  errors:  failure  to 
detect  the  presence  of  a  target  or  targets,  failure  to  distinguish 
between  target  and  non-target  objects,  detecting  an  object  where  there 
was  none  (inventive  errors),  failure  to  translate  accurately  target 
locations  in  comparing  photos,  and  failure  to  make  the  correct  same- 
different  discrimination. 

Change  Detection  with  Target  Identification.  At  this  level,  the 
interpreter  was  required  to  identify  the  target  as  belonging  to  one  of 
nine  target  categories  listed  on  the  Response  Code  Sheet  (Figure  2). 
Having  detected  and  identified  a  target,  he  proceeded  to  assign  the 
change  status  designation  as  at  the  detection  level.  In  addition,  where 
a  target  was  uniquely  identifiable,  the  change  status  'Gloved"  was  appli¬ 
cable  if  the  target  had  been  repositioned  within  the  same  general  area. 
All  sources  of  erroneous  change  status  assignment  cited  above  were  appli¬ 
cable  to  the  identification  level  of  response,  plus  misidentification  of 
detected  targets. 

The  method  of  evaluating  interpreter  performance  placed  maximum 
emphasis  on  the  change  detection  aspect  of  the  task.  In  scoring,  a 
change  status  assignment  was  counted  as  correct  only  when  the  total 
present-absent  and  same-different  comparison  had  been  accurately  per¬ 
formed  . 
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RESPONSE  CAT EGORIES 


CONFIDENCE  LEVELS 


CHANGE  STATUS 


1 .  Tanks 

(includes  M4l,  M48,  M60) 


2.  SP  Howitzers  &  Guns 

(includes  M42,  M44,  M53, 
M35,  M56,  M108,  M109) 


C  --  Certain  U  --  Unchanged 

FS  --  Fairly  Sure  M  --  Moved 

D  --  Somewhat  (within  area) 

Doubtful  N  --  New 

G  --  Best  Guess  G  --  Gone 

?  --  Something  R  --  Replaced 


3 .  Armored  Personnel 
Carriers 

( includes  M59,  M113, 

M114) 

4.  Trucks  -  3/4  Tons  &  Less 
(includes  M37,  M38,  M151) 

5.  Trucks  -  2-1/2  Tons  & 
Greater 

(includes  M34,  M35,  M36, 
M41 ,  M54,  M55,  M40) 

6.  Trailers  4  Semi-Trailers 
( includes  Cargo,  Water, 
Gas,  Tank  Transport) 


7.  Engineering  Equipment 
(includes  Cranes, 
Graders,  Scoops, 
Tractors,  etc.) 

8.  Aircraft 

( includes  fixed  & 
rotating  wing) 


9 .  Others 

( includes  Wrecker,  M543  4 
M62;  Truck  Tractors, 

M52  4  M123;  Recovery 
Vehicle,  M88;  does  not 
include  Tents,  buildings, 
stock  piles,  roads, 
railroads,  rivers) 


Figure  2.  Reproduction  of  Response  Code  Sheet  used  in  the  experiment 


Test  Materials  and  Equipment 

Photography .  From  photography  available  in  BESRL's  Technical  Support 
Branch  Film  Library,  negatives  for  ten  sets  of  comparative- cover  aerial 
photographs  of  Camp  Drum,  New  York  were  selected.  Selection  was  based 
primarily  on  number  of  targets  and  target  types  within  each  pair  and  the 
resultant  total  representation  in  the  final  test  sample.  Number  and  dis¬ 
tribution  of  targets  and  types  of  change  contained  in  each  photo  pair  are 
shown  in  Table  A-2  of  the  Appendix.  Scale,  image  quality,  dominant  ter¬ 
rain  characteristics,  and  amount  of  overlap  were  not  systematically  con¬ 
trolled  (see  Table  A-3).  Positive  black-and-white  transparencies  in  a 
9"  by  9"  format  were  prepared  in  sufficient  quantity  to  provide  each  of 
the  64  subjects  with  the  required  ten  sets  of  imagery  for  analysis  under 
their  assigned  conditions. 

Materials .  A  Response  Code  Sheet  (Figure  2)  was  prepared  to  enumer¬ 
ate  and  describe  the  nine  target  response  categories  to  be  used  in  re¬ 
porting  the  identification  of  targets  detected,  confidence  levels  to  be 
used  in  association  with  each  identification,  and  the  five  types  of 
change  status.  The  confidence  rating  was  included  in  order  to  simulate 
more  closely  operational  procedures  where  it  is  common  practice  of  inter¬ 
preters  to  rate  their  confidence  on  the  information  they  report.  An 
Immediate  Report  Form  (Figure  3)  was  provided  with  each  photo  pair  to 
standardize  target  reporting  and  identify  subject,  experimental  condition, 
and  stimulus  material. 

Each  subject  was  issued  a  9"  by  9"  transparent  celluloid  target 
localization  grid,  ruled  off  into  144  consecutively  numbered  cells  3/4- 
inch  square.  When  overlaid  on  the  most  recent  coverage  of  each  compara¬ 
tive  photo  pair,  the  grid  provided  a  numerical  reference  system  for  re¬ 
porting  the  location  of  the  detected  targets.  Interpreter  keys  for  the 
identification  of  U.  S.  military  vehicles  were  made  available  to  all 
subjects  for  study  prior  to  the  test  session  and  for  reference  as  needed 
during  the  test.  All  additional  supplies--grease  pencils  for  annotating 
photos,  tracing  paper  for  preparing  overlays,  etc. --were  provided  by  the 
experimenters . 

Display  and  Viewing  Equipment .  Regulation  government- issue 
variable-illumination  swivel-top  light  tables  were  provided  for  viewing 
the  positive  transparencies.  Each  subject  had  his  own  photo- interpreta¬ 
tion  kit  from  which  he  was  free  to  use  whatever  aids  he  desired-- 
roagnifiers,  for  example — in  performing  the  assigned  task. 
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U.  S.  ARMY  BEHAVIORAL  SCIENCE  RESEARCH  LABORATORY 
WASHINGTON,  D.  C.  2031 5 


IMMEDIATE  REPORT  FORM 

Name _ Man  No.  _ 

Date  _  Perf.  Meas.  No. 


Figure  3.  Reproduction  of  Immediate  Report  Form  used  in  the  experiment 


Experimental  Procedure 

As  each  experimental  group  arrived  at  the  test  site  for  the  first 
time,  subjects  were  given  a  short  briefing  on  the  objectives  and  proce¬ 
dures  of  the  experimentation  in  which  they  were  about  to  participate. 

They  were  given  a  sufficient  amount  of  information  concerning  the  vari¬ 
ables  under  investigation  to  make  the  task  meaningful  to  them,  and  the 
implications  and  applications  of  their  efforts  were  emphasized  to  instill 
a  positive  motivational  set  (see  Instructions  to  Subjects  in  Appendix  B). 
A  test  packet  containing  materials  for  the  test  (photos  and/or  reference 
materials,  ten  Immediate  Report  Forms,  a  Response  Code  Sheet,  a  transpar¬ 
ent  reference  grid,  a  copy  of  the  PI  Key  to  U.  S.  Military  Vehicles, 
tracing  paper,  and  grease  pencils)  was  then  distributed  to  each  subject. 

Subjects  were  instructed  on  l )  use  of  the  Immediate  Report  Form  to 
record  their  responses  (sequential  numbering  of  detected  targets,  coding 
of  target  identification  by  target  class,  reporting  of  target  location 


-  9  - 


by  grid  cell  designation,  code  designation  of  each  test  photo,  basis  for 
establishing  and  recording  change  status  judgments,  and  indication  of 
lapsed  interpretation  time);  2)  meaning  and  application  of  each  of  the 
items  on  the  Response  Code  Sheet;  and  3)  procedure  to  follow  in  preparing 
the  annotated  photos  (e.g.,  circle  with  grease  pencil  and  number  con¬ 
secutively  all  targets  detected  on,  and  missing  from,  the  later  imagery) 
and  in  producing  the  situational  overlays  (e.g.,  include  all  detected 
targets  and  enough  surrounding  detail  to  permit  subsequent  matching  with 
other  imagery) . 

The  32  subjects  serving  as  experienced  interpreters  participated  in 
two  test  sessions,  whereas  the  no-experience  groups  participated  in  only 
one.  In  the  first  session,  the  experienced  subjects  produced  annotated 
photographs,  situational  overlays,  and  written  reports  (exclusive  of 
change  status  determination)  for  the  earlier  cover  (P^).  No  time  limit 

was  imposed.  In  the  second  session  for  those  designated  experienced 
subjects,  which  took  place  24  hours  after  the  first,  subjects  were  re¬ 
sponsible  only  for  annotating  the  later  imagery  (P^)  and  preparing  an 

Immediate  Report  Form  for  each  photo.  No  overlay  was  required.  Inspec¬ 
tion  time  was  limited  to  either  5  or  15  minutes  per  picture.  Subjects 
were  also  required  to  make  change  status  statements  for  each  target  by 
making  reference  to  the  information  provided  (or  remembered).  The 

no-experience  interpreters,  having  only  one  session,  were  required  to 
produce  annotated  photos  and  Immediate  Report  Forms  similar  to  those 

being  prepared  by  the  experienced  subjects  in  their  second  test  session. 

The  material  supplied  to  the  subjects  for  comparison  with  the  P^ 

imagery  varied  with  experimental  condition.  The  experienced  subjects  who 
were  designated  to  work  with  annotated  photo  imagery  received  the  annotated 
photos  which  they  themselves  had  prepared  in  the  previous  test  session. 

The  experienced  subjects  working  under  the  overlay- and -report  condition 
received,  along  with  the  imagery,  the  10  overlays  and  Iinnediate  Report 

Forms  they  had  themselves  prepared  the  day  before.  From  materials  pre¬ 
pared  by  the  remaining  24  experienced  subjects  in  their  first  session, 
the  annotated  imagery  and  overlay-report  reference  material  were  selected 
for  the  no-experience  subjects.  Annotated  photos  and  overlays  plus  re¬ 
port  were  randomly  selected  and  paired  with  the  appropriate  P^  image. 

Except  for  the  computer-comparison  groups,  each  no-experience  subject 
received  comparative  P^  materials  prepared  by  a  randomly  selected  10  out 

of  the  24  experienced  subjects.  Input  for  each  of  the  10  scenes  an 

individual  was  to  analyze  had  been  prepared  by  a  different  experienced 
subject . 

Questions  relating  to  the  procedures  were  answered  and  the  testing 
was  conducted  with  three  experimenters  continuously  monitoring  the  sub¬ 
jects  to  assure  adherence  to  the  time  limit  imposed.  Subjects  were  given 
the  tests  in  their  normal  classrooms  during  regular  duty  hours.  Four  group 
sessions  were  held  with  up  to  20  subjects  participating  in  each  session. 
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Scoring 


Data  Obtained .  The  units  of  information  obtained  by  the  reporting 
procedure  described  included: 

1,  A  target  position  report  localizing  each  of  the  detected  targets 
within  one  of  the  144  cells  dividing  the  photograph. 

2.  A  coded  identification  of  the  target  class  (one  of  nine  cate¬ 
gories)  to  which  each  detected  target  was  judged  to  belong. 

5.  A  coded  statement--Unchanged  (U),  Moved  (m),  Gone  (G),  New  (N), 
or  Replaced  (R)--of  the  judged  status  of  each  reported  target  as  deter¬ 
mined  from  comparison  of  the  two  image  samples  or  image  sample  and 
reference  data. 

Individual  responses  were  scored  by  comparing  each  report  of  a 
target  location,  identification,  and  change  status  with  "interpreter 
truth"  data  provided  by  a  team  of  experts  in  the  U.  S.  Army  Behavioral 
Science  Research  Laboratory.  The  interpreter  truth  for  each  target  on 
a  photo  pair--target,  location,  and  change  status--was  encoded  and  tran¬ 
scribed  on  punched  cards.  The  responses  to  these  targets  as  reported  by 
each  subject  were  similarly  encoded  and  transcribed  from  the  Inmediate 
Report  Forms  to  the  same  cards.  Inventive  errors  were  identified  and 
entered  separately.  Comparison  of  the  transcribed  responses  with  inter¬ 
preter  truth  yielded  quantitative  data  on  1)  number  of  targets  correctly 
detected  (  and  by  subtraction,  number  omitted);  2)  number  of  correctly 
detected  targets  which  were  correctly  identified  and  the  number  incorrectly 
identified;  5)  number  of  correct  and  incorrect  change  status  assignments; 
and  4)  number  of  inventive  errors  committed  along  with  number  of  erroneous 
identifications  and  change  status  designations  associated  with  the  inven¬ 
tive  error. 


Evaluative  Measures 


From  the  response  data  available,  evaluations  of  interpreter  perform¬ 
ance  were  derived  for  application  in  assessing  the  effects  of  the  inde¬ 
pendent  variables.  Separate  measures  of  interpreter  performance  were 
computed  for  change  status  detection  without  identification  of  target  and 
for  change  status  detection  requiring  target  identification. 

Change  Status  Completeness.  The  completeness  of  interpretation  is 
the  ratio  of  the  number  of  correct  chan0e  status  responses  to  th  1  total 
number  of  correct  change  status  responses  possible.  In  equation  form: 


COMPLETENESS 


Number  of  Correct  Change  Status  Responses 
Total  Possible  Correct  Change  Status  Responses 
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Change  Status  Accuracy.  The  accuracy  of  Interpretation  is  the  ratio 
of  the  number  of  correct  change  status  responses  to  the  total  number  of 
change  status  responses  (both  correct  and  incorrect)  reported  by  the 
Interpreter  subject.  In  equation  form: 


ACCURACY  - 


Number  of  Correct  Change  Status  Responses _ 

Total  Number  of  Change  Status  Responses  Reported 


To  provide  comparable  performance  measurement  units  from  photo  pairs 
containing  different  numbers  of  targets,  the  completeness  and  accuracy 
scores  of  each  subject  were  averaged  over  all  photo  pairs  viewed  under 
each  experimental  condition  to  yield  mean  performance  figures  for  each 
treatment  condition. 


RESULTS 


Completeness  of  Change  Detection 

Change  Detection  without  Target  Identification.  Table  2  presents 
the  mean  completeness  scores  for  the  change  status  responses  when  target 
identification  was  not  required.  Signlf leant  main  effect  differences 
were  found  for  all  independent  variables  (see  analysis  of  variance  summary 
table  4  in  Appendix  A). 

.The  experienced  groups  generally  performed  considerably  better  than 
did  the  no-experience  groups  (56$  and  48$,  respectively).  The  simulated 
computer  comparison  condition,  relying  on  separate  interpretations  of  the 
two  comparative  cover  photos  and  determination  of  change  status  by  a 
computer  method,  was  best  (67$  completeness),  followed  by  comparative 
cover  interpretation  of  later  cover  with  an  annotated  photograph  and  with 
an  unannotated  photograph  (51$  and  50$,  respectively)  of  earlier  cover. 

The  poorest  performance  was  with  the  procedure  relying  on  written  reports 
and  photo  overlays  for  comparison  with  the  photo  representing  the  later 
coverage  (38$  completeness). 

As  anticipated,  additional  interpretation  time  resulted  in  more 
complete  change  detection  reports.  Of  the  possible  change  status  assign¬ 
ments,  61$  were  provided  after  15  minutes  as  compared  with  only  43$  after 
5  minutes. 

Two  significant  interactions  were  indicated:  l)  Experience  with  the 
area  by  Method  of  Comparison,  and  2)  Experience  by  Method  of  Comparison 
by  Time.  The  Experience  by  Comparison  Method  interaction  suggests  that, 
whereas  both  the  experienced  and  the  no-experience  interpreters  performed 
about  equally  well  under  the  computer  comparison  where  the  task  was  essen¬ 
tially  one  of  normal  single  frame  interpretation  not  involving  direct 
change  detection  (mean  completeness  scores  were  67$  and  68$),  for  the 
other  comparison  methods  the  experienced  group  did  consistently  better 
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than  the  no-experience  group.  This  result  was  not  unexpected,  since 
the  experienced  group  had  the  advantage  of  working  with  reference  mate¬ 
rials  they  themselves  had  generated  whereas  the  no-experience  group 
worked  with  unfamiliar  early  coverage  materials  generated  by  others. 

The  Interpreter  Experience  by  Comparison  Method  by  Time  interaction  may 
result  from  the  fact  that  the  experienced  interpreters  performed  substan¬ 
tially  better  with  short  inspection  time  using  their  own  overlay  plus 
written  report  materials  than  did  the  no-experience  interpreters  using 
materials  prepared  by  others.  This  advantage  was  evident  only  in  the 
overluy  plus  written  report  (OR)  method  under  the  15-minute  time  limit. 
The  difference  in  completeness  between  the  two  experience  groups  under 
the  15-minute  work  time  with  the  OR  method  may  derive  from  the  fact  that 
the  no-experlence  interpreter  cannot  correct  or  supplement  the  early  re¬ 
port  since  he  has  not  seen  the  earlier  cover,  whereas  the  experienced 
interpreter,  comparing  current  imagery  with  a  report  that  he  himself 
prepared,  can  supplement  his  OR  comparison  with  his  recollections  of  the 
earlier  imagery. 

One  additional  finding  is  of  interest:  Although  a  significant  in¬ 
teraction  for  Time  by  Comparison  Method  was  not  found,  the  average  com¬ 
pleteness  for  the  computer  comparison  method  increased  only  very  slightly 
with  an  increase  in  time  (from  66$  to  68$)  relative  to  those  for  the 
other  conditions.  It  would  appear  that  the  computer  comparison  method 
is  generally  superior  to  the  others,  with  very  little  gain  to  be  derived 
from  adding  more  inspection  time. 


Table  2 

MEAN  COMPLETENESS  SCORES  FOR  CORRECT  TARGET  CHANGE  WHEN 
TARGET  IDENTIFICATION  WAS  NOT  REQUIRED 


Groups 

No- 

■Experience 

Experienced 

Combined 

Reference 

Materials 

Time 

Limits 

Time 

Limits 

Time 

Limits 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

CC 

.663 

.686 

.675 

.655 

.683 

.668 

.658 

.684 

.67I 

UP 

.223 

.660 

.441 

.512 

.610 

.561 

.368 

.635 

.501 

AP 

.318 

.619 

.468 

.489 

.621 

•555 

.404 

.620 

.512 

OR 

.244 

-83 

.314 

.322 

.590 

.456 

.283 

.486 

.385 

Means 

.362 

.587 

.475 

.494 

.626 

•  560 

.428 

•  606 

_ 1 

.517 
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Change  Detection  with  Target  Identification.  The  results  for  this 
analysis  were  essentially  similar  to  those  for  performance  at  detection 
level,  except  that  completeness  was  generally  lower--a  not  surprising 
finding,  since  the  added  requirement  for  target  identification  was  con¬ 
tingent  upon  target  detection.  In  addition  to  each  main  effect,  all  in¬ 
teractions  were  significant  (see  Appendix  Table  A-5).  Table  3  presents 
the  mean  performance  values  for  this  response  level. 

The  experienced  group  performed  generally  better  than  the  no¬ 
experience  group  (40$  vs  39$).  The  computer  comparison  condition  re¬ 
sulted  in  better  performance  than  the  two  photo-comparison  conditions, 
which  in  turn  were  better  than  the  overlay  plus  report  condition  (com¬ 
pleteness  scores  were  48$,  38$,  37$,  and  28$,  respectively).  The  15- 
minute  time  allowance  resulted  in  more  complete  performance  than  the 
5-minute  allowance  (44$  vs  32$). 

The  Experience  by  Comparison  Method  interaction  again  indicated 
that  experienced  interpreters  performed  significantly  better  than  no¬ 
experience  interpreters  when  they  use  comparison  materials  which  they 
have  themselves  generated.  The  Experience  by  Time  interaction  was  also 
found  to  be  significant.  Experienced  interpreters  performed  better  in 
the  5“n'inute  time  period  than  did  the  no-experience  interpreters,  but 
there  was  no  such  difference  with  the  15-minute  period.  Mean  complete¬ 
ness  scores  for  experienced  and  no-experience  interpreters  were  37$  and 
27$,  respectively,  for  5-minutes  of  interpretation  and  44$  and  44$  for 
the  15-minute  time  period. 

The  Time  by  Comparison  Method  interaction  indicated  that  additional 
time  is  helpful  under  all  conditions  other  than  the  computer  comparison, 
under  which  performance  was  slightly  degraded  with  the  longer  inspection 
time.  Completeness  under  this  condition  was  52$  with  the  5-minute  period 
and  declined  to  46$  with  the  15-minute  period. 

The  Experience  by  Comparison  Method  by  Time  interaction  again  in¬ 
dicated  that  experienced  interpreters  performed  better  working  with 
materials  they  had  generated  than  did  the  no-experience  interpreters 
working  with  reference  materials  generated  by  others  under  the  5-minute 
time  allowance.  Under  the  15-minute  time  period,  the  experienced  inter¬ 
preter  reached  a  much  higher  level  of  completeness  than  his  no-experience 
counterpart  for  the  OR  comparison  method  since  he  could  supplement  the 
overlay  locations  and  written  report  descriptions  with  his  personal  recol¬ 
lections,  This  possible  memory  advantage  was  not  evident  with  the  Com¬ 
puter  Comparison  Method  where  the  experienced  interpreter  was  much  less 
complete  after  15  minutes  than  is  his  no-experience  peer. 

To  summarize  the  completeness  performance  results  in  the  stud)',  a 
computer  comparison  procedure  similar  to  the  one  simulated  here  in  which 
an  objective  and  automated  comparison  of  two  independent  inputs  is  per¬ 
formed,  can  effectively  lead  to  the  production  of  more  timely  and  com¬ 
plete  change  status  reports.  It  is  equally  clear  that  previously  estab¬ 
lished  familiarity  with  the  target  area  and  reporta  from  earlier 
coverage  is  particularly  advantageous  when  reporting  time  is  a  critical 
factor. 
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Table  3 


MEAN  COMPLETENESS  SCORES  FOR  TARGETS  DETECTED,  CORRECTLY  IDENTIFIED, 
AND  ASSIGNED  CORRECT  CHANGE  STATUS 


Groups 

No-Experience 

Experienced 

Combined 

Reference 

Materials 

Time 

Limits 

Time 

Limits 

Time 

Limits 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

CC 

.505 

.524 

.515 

.524 

.385 

.455 

.515 

.455 

.485 

UP 

.193 

.471 

•  352 

.372 

.467 

.420 

.282 

.469 

•  376 

AP 

.198 

.492 

.345 

.362 

.421 

.392 

.280 

.457 

.368 

OR 

.177 

.251 

.214 

.228 

.483 

.356 

.203 

.367 

.285 

Means 

.268 

•  435 

•  351 

.372 

.439 

.405 

.320 

.437 

.378 

Accuracy  of  Change  Detection 

Change  Detection  without  Target  Identification.  Directional  trends 
for  accuracy  of  change  detection  were  in  fair  accord  with  those  for  com¬ 
pleteness  although  only  two  main  eff ects--Experience  and  Method  of 
Comparison — were  found  to  be  significant  at  the  .05  level  (Appendix 
Table  A-6).  As  in  the  case  of  completeness  scores,  the  experienced 
interpreters  tended  to  perform  better  than  the  no-experience  interpreters 
■when  all  viewing  conditions  were  considered.  Mean  accuracy  score  of  ex¬ 
perienced  interpreters  was  79$  as  opposed  to  only  72$  for  no-experience 
interpreters  (Table  4).  In  regard  to  Comparison  Method,  the  computer- 
comparison  condition  provided  80$  accuracy,  roughly  equivalent  to  that 
attained  with  either  the  annotated  photos  (80$)  and  the  unannotated 
photos  (75$).  The  overlay  and  report  comparison  resulted  in  performance 
accuracy  considerably  less  than  any  of  the  other  three  (66$). 

Only  with  the  accuracy  index  was  there  some  indication  (for  no¬ 
experience  interpreters  at  least)  that  when  time  pressure  exists  there 
is  an  advantage  to  working  with  annotated  as  opposed  to  unannotated 
imagery  (74$  vs  55$  accuracy). 
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Table  4 


MEAN  ACCURACY  SCORES  FOR  CORRECT  TARGET  CHANGE  STATUS  RESPONSES 


i 

I  Groups 

No -Experience 

Experienced 

Combined 

Reference 

Materials 

Time 

Limits 

Time 

Limits 

Time 

Limits 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

CC 

CD 

00 

• 

•799 

.822 

.818 

.746 

.782 

.832 

.772 

.802 

UP 

.549 

O 

K-\ 

00 

.690 

•792 

00 

.81? 

.671 

CM 

rO 

CO 

• 

•751 

AP 

.739 

.824 

.78I 

.824 

.804 

.814 

.781 

.814 

•798 

OR 

.589 

•365 

•577 

.654 

.859 

•746 

.621 

.702 

.661 

Means 

.680 

•755 

.717 

•772 

.806 

.789 

•726 

.780 

•753 

Change  Detection  with  Target  Identification.  As  expected,  the  re¬ 
sults  of  this  analysis  were  essentially  the  same  as  those  obtained  for 
responses  without  target  identification,  although  performance  was  some¬ 
what  lower  (Table  5).  The  analysis  of  variance  (Appendix  Table  A-7) 
indicated  that  none  of  the  main  effect  differences  were  sufficiently 
reliable  to  be  of  statistical  significance.  The  triple  interaction  did 
reach  significance  at  the  .05  level.  Time  and  Experience  produced  differ¬ 
ent  results  with  the  comparison  methods  employed.  Under  the  5-roinute  time 
limit  the  experienced  interpreters  were  more  accurate  than  the  no-experience 
interpreters.  Under  the  15-minute  time  limit,  this  greater  accuracy  dif¬ 
fered  with  experience  level  and  comparison  method.  The  more  extreme 
differences  were  with  the  computer  comparison  and  the  overlay  plus  report 
methods.  The  experienced  men  improved  in  accuracy  with  the  OR  method 
whereas  the  no-experience  men  were  less  accurate  under  the  15-minute  time 
limit.  However,  with  the  computer  comparison  method,  the  accuracy  of  the 
experienced  men  deteriorated  markedly  under  the  15-minute  time  limit 
whereas  the  accuracy  of  the  no-experience  men  declined  only  a  small  amount. 
As  stated  before,  the  experienced  interpreter  appears  to  have  some  residual 
recollection  of  the  imagery  he  interpreted  previously  that  he  can  use  to 
supplement  the  overlay  plus  written  report  method.  With  the  computer  com¬ 
parison  method,  these  recollections  appear  to  have  worked  against  him, 
since  he  had  no  documents  which  showed  his  interpretations  of  the  earlier 
imagery. 


16 


Table  5 


MEAN  ACCURACY  SCORES  FOR  TARGETS  DETECTED,  CORRECTLY  IDENTIFIED, 
AND  ASSIGNED  CORRECT  CHANGE  STATUS 


Groups 

No- 

■Experience 

Experienced 

Combined 

Reference 

Materials 

Time 

Limits 

Time 

Limits 

Time 

Limits 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

5  Min 

15  Min 

Means 

CC 

.620 

.588 

.604 

.650 

.421 

.536 

.635 

•505 

•  570 

UP 

.433 

•579 

•  506 

-575 

.621 

.598 

.504 

.600 

•  552 

AP 

.470 

.633 

•551 

.581 

•  540 

•  560 

.525 

.586 

•  556 

OR 

.403 

.366 

.385 

.417 

.668 

.543 

.410 

.517 

.464 

Means 

.481 

.541 

-511 

•  536 

.563 

•559 

.519 

.552 

•555 

Analysis  of  Responses  in  Terms  of  Change  Status 

For  readers  who  are  interested  in  investigating  interpreter  response 
tendencies  in  greater  depth,  change  detection  data  in  relation  to  type 
of  change  is  presented  in  more  detail  in  Appendix  C.  Included  are  summary 
data  showing  the  distribution  of  correct  and  incorrect  responses  for  each 
type  of  change  and  the  frequency  with  which  inventive  errors  were  made  in 
each  type  (Table  C-l).  A  second  table  (Table  C-2)  shows  the  distribution 
of  erroneous  change  status  assignments  across  the  incorrect  change  cate¬ 
gories.  To  facilitate  comparison  between  change  categories  having  differ¬ 
ent  numerical  bases,  the  frequency  data  have  been  converted  to  percentages. 
A  third  table  gives  the  raw  score  distributions.  Some  cursory  conclusions 
and  speculative  hypothesizing  about  causal  factors  are  offered. 


CONCLUSIONS 

In  general,  the  method  of  deriving  target  change  status  reports  by 
having  interpreters  analyze  the  earlier  and  later  frames  separately  and 
submit  these  inputs  to  a  computer  for  comparison  and  change  status  de¬ 
termination  appears  promising.  This  procedure  seems  to  provide  change 
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status  reports  which  are  both  complete  and  accurate,  particularly  when 
interpretation  time  is  limited.  This  is  not  to  say  that  conventional 
comparative  cover  interpretation  involving  the  comparison  of  an  early 
and  a  late  photo  should  be  discontinued,  since  this  procedure  is  still 
advantageous  for  certain  types  of  comparison  such  as  target  verifica¬ 
tion  and  evaluation  of  construction  progress.  The  range  of  interpreter 
activities  for  which  a  computerized  procedure  would  be  profitable  re¬ 
mains  to  be  established,  as  does  also  the  optimal  encoding  format  for 
input  of  the  required  information. 

In  the  absence  of  computer  comparison  capability,  conventional 
comparative  cover  interpretation  involving  the  simultaneous  use  of  both 
early  and  late  cover  remains  the  most  useful  procedure.  There  does  not 
appear  to  be  any  consistent  and  demonstrable  advantage  to  be  derived 
from  providing  interpreters  with  photo  pairs  which  include  annotations 
resulting  from  an  earlier  analysis. 

Change  detection  appears  to  be  somewhat  superior  when  the  inter¬ 
preter  is  already  familiar  with  the  target  area  being  viewed.  The 
Interpreter  familiar  with  the  area  is  at  particular  advantage  when  time 
is  extremely  critical.  However,  lack  of  familiarity  can  be  compensated 
for  by  Increasing  the  time  allotted  to  interpretation.  For  maximum 
efficiency  of  operation,  interpreters  familiar  with  a  particular  area 
should  be  assigned  to  perform  subsequent  comparative  cover  analyses  of 
that  area. 

In  the  event  ample  interpretation  time  is  available,  it  makes  little 
difference  whether  interpreters  are  familiar  with  the  area,  nor  does  the 
type  of  reference  material  provided  greatly  affect  the  completeness  or 
accuracy  of  the  change  status  report  generated.  Non-pictorial  (schematic 
and/or  verbal)  reference  material  appears  to  be  a  poor  basis  on  which  to 
make  change  status  determinations.  If  such  material  must  be  used,  it  is 
desirable  that  the  Interpreter  use  material  he  has  himself  prepared. 
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APPENDIX  A -SUPPLEMENTARY  STATISTICAL  TABLES 


Table  A-l 

MATCHING  OF  TEST  GROUPS  OF  4  EACH  ON  THE  BASIS  OF 
GENERAL  TECHNICAL  APTITUDE  AREA  SCORES 


Group  No. 

Mean  General  Technical 
Aptitude  Area  Score 
(N  -  4) 

Standard 

Deviation 

1 

118.7 

14.2 

2 

118.0 

13.7 

3 

11Y.2 

11.6 

4 

117.5 

11.4 

5 

U7.0 

14.9 

6 

118.2 

16. 9 

7 

126.2 

20.6 

8 

117.5 

9.0 

9 

118.0 

7.0 

10 

117.5 

11.8 

11 

117.7 

9.6 

12 

115.0 

6.0 

13 

116.0 

9.7 

14 

118.7 

6.8 

15 

117.2 

8.6 

16 

116  .7 

8.3 

2: 


Table  A-3 


DESCRIPTION  OF  UNCONTROLLED  IMAGE  PARAMETERS 


Photo 

Pair 

Photo 

Identification 

Number 

Date 

Quality 

Scale 

Perc ent 
Common 
Area 

1 

D- 18/27 

6  Sept  62 

C 

3500 

95 

D- 50/30 

5  Sept  64 

G 

2450 

2 

D- I9/67 

6  Sept  62 

G 

5000 

85 

D-20/43 

7  Sept  62 

G 

5200 

5 

D-21/32 

10  Sept  62 

G 

900 

100 

D-25/58 

12  Sept  62 

F 

2100 

4 

S-00/06 

25  July  62 

G 

2900 

85 

D-45/40 

8  Sept  64 

F 

3100 

5 

S-00/25 

25  July  62 

G 

2900 

75 

D-45/62 

8  Sept  64 

G 

3300 

6 

D-O6/I5 

9  Sept  62 

G 

6000 

100 

D- 24/26 

12  Sept  62 

G 

1900 

7 

D-18/80 

6  Sept  62 

G 

3800 

100 

D-41/14 

7  Sept  64 

F 

2900 

8 

D- 37/54 

3  Sept  64 

F 

3500 

85 

D-45/05 

8  Sept  64 

G 

2900 

9 

D-41/82 

7  Sept  64 

G 

3000 

90 

D- 52/26 

7  Sept  64 

G 

1500 

10 

D- 54/85 

8  Sept  64 

F 

4600 

100 

D- 37/48 

3  Sept  64 

G 

3500 

Analysis  of  Variance  Summary  Tables 


Table  A-4 


COMPLETENESS  SCORES  FOR  CORRECT  TARGET  CHANGE  STATUS  RESPONSES 
WITHOUT  TARGET  IDENTIFICATION 


Source  of 
Variance 

Sum  of 
Squares 

Degrees  of 
Freedom 

Mean 

Squares 

F-Ratio 

Experience  (A) 

.117 

1 

.117 

13.34* 

Time  (  B) 

.508 

1 

.508 

57  .88* 

Reference  (C) 

.605 

3 

.222 

25.27* 

AxB 

.035 

1 

.035 

3.94  NS 

BxC 

.051 

3 

.017 

1.96  NS 

AxC 

.133 

3 

.044 

5.05* 

AxBxC 

•125 

3 

.042 

4.75* 

Residual 

.421 

48 

.009 

Total 

2.056 

63 

*P  <  .01 


Table  A- 5 


COMPLETENESS  SCORES  FOR  CORRECT  TARGET  CHANGE  STATUS  RESPONSE 
WITH  TARGET  IDENTIFICATIONS 


Source  of 
Variance 

Sum  of 
Squares 

Degrees  of 
Freedom 

Mean 

Squares 

F-Ratio 

Experience  (A) 

.047 

1 

.047 

5.28* 

Time  (  B) 

.219 

1 

.219 

24 .82** 

Reference  (C) 

.323 

3 

.108 

12.20** 

AxB 

.039 

1 

.039 

4.45* 

BxC 

.088 

3 

.029 

3.32* 

AxC 

.169 

3 

.056 

6.38** 

AxBxC 

.108 

3 

.036 

4.07* 

Residual 

.423 

48 

.009 

Total 

1 .415 

63 

*P  <  .05 
**P  <  .01 
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Analysis  of  Variance  Sutrmary  Tables 


Table  A-6 


ACCURACY  SCORES  FOR  CORRECT  TARGET  CHANGE  STATUS  RESPONSES 
WITHOUT  TARGET  IDENTIFICATION 


Source  of 

Sum  of 

Degrees  of 

Mean 

Variance 

Squares 

Freedom 

Squares 

F-Ratio 

Experience  (A) 

.081 

1 

.081 

5.37* 

Time  (B) 

.047 

1 

.04? 

3-09 

Reference  ( C) 

.204 

3 

.068 

4.50** 

AxB 

.006 

1 

.006 

.42 

BxC 

.105 

3 

.035 

2.32 

AxC 

.103 

3 

.034 

2.25 

AxBxC 

.106 

3 

.035 

2.33 

Residual 

.728 

48 

.015 

Total 

1.138 

63 

*P  <  .05 
**P  <  .01 

ACCURACY 

Table  A-7 

SCORES  FOR  CORRECT  CHANGE  STATUS  RESPONSES 

WITH  TARGET  IDENTIFICATION 

Source  of 

Sum  of 

Degrees  of 

Mean 

Variance 

Squares 

Freedom 

Squares 

F-Ratio 

Experience  (A) 

.036 

1 

.036 

1.95 

Time  (B) 

.018 

1 

.018 

•95 

Reference  ( C) 

.113 

3 

.037 

2.00 

AxB 

.011 

1 

.011 

.60 

BxC 

.116 

3 

.039 

2.07 

AxC 

.148 

3 

.049 

2.63 

AxBxC 

.162 

3 

.054 

2.88* 

Residual 

.900 

48 

.019 

Total 

1.505 

63 

*P  <  .Ofj 
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APPENDIX  B 


CHECKLIST  FOR  INSTRUCTIONS  TO  SUBJECTS 


I,  Orientation  Briefing  (stressing  the  need  for  empirical  data  which 
will  significantly  advance  photo-reconnaissance  intelligence 
"state-of-the-art") 

II.  Introduction  of  BESRL  and  Boeing  Test  Personnel 

III.  Brief  Review  of  Research  Program  and  Test  Plan 


A.  Three  experiments  planned  for  this  program 

B.  Experiment  I  involves  16  different  experimental  conditions 

1.  Primarily  to  investigate  parameters  influencing  information 
extraction  (accuracy,  completeness,  and  time)  from  aerial 
photography. 

2.  Major  current  concern  is  witfi  change  detection  with 
comparative  cover  photography. 

3.  Performance  measures  to  be  obtained  during  test  sessions 
are  not  intended  to  reflect  individual  aptitudes  or  skills 
but  serve  as  representative  data  base  for  all  Pis.  Scores 
achieved  on  these  tests  will  not  affect  class  grade  at 
USAINTS. 

4.  Operational  Implications:  Variety  of  field  operations 
conditions  represented  in  tast  program  (i.e.,  one  man  vs 
team  approach  as  reflected  in  the  conditions  where  S's  may 
or  may  not  have  previously  interpreted  P^,  types  of  re¬ 
duced  or  symbolic  reference  material  to  be  stored  and 
retrieved  for  subsequent  comparisons,  delineation  of  time 
(or  load)  requirements  to  assure  maximum  efficiency  of 
intelligence  systems). 


IV.  Distribution  of  Teat  Materials  (Photo  packets,  grids,  etc.)  and  PI 
Keys 


V.  General  Test  Procedures,  Instructions,  and  Ground  Rules 
A.  Alignment  of  Imagery  and  Grid 

1.  Pp  is  always  to  be  placed  on  the  light  table  so  that  the 

identification  number  is  located  In  the  lower  right  corner 
of  the  imagery  and  reads  normally.  The  same  is  true  for 
"P^  only"  when  initially  interpreted  by  the  "Prior 

Interpretation"  group. 


2.  The  reference  grid  is  to  be  superimposed  over  the  trans¬ 
parency,  so  that  the  lower  left  corner  of  each  is  coincident. 

B.  Optical  Aids 

1.  Any  instruments  normally  contained  in  the  standard  issue 

PI  kit  may  be  used  to  assist  in  the  inspection. 

C.  Recording  of  Responses 

1 .  Immediate  Report  Form. 

a.  Col.  I.  Each  item  detected  must  be  numbered  consecu¬ 
tively  in  this  column  and  correspond  to  the  numbers 
which  are  associated  with  each  of  these  targets  on  the 
annotated  photograph  and  overlay. 

b.  Col.  II.  The  .terns  detected  need  not  be  identified  by 
name  but  the  number  corresponding  to  the  appropriate 
target  class  (as  shown  on  "Response  Categories"  sheet) 
must  be  placed  in  third  column.  If  several  of  the  same 
type  targets  are  located  in  the  same  iimediate  area,  a 
single  entry  showing  quantity  and  class  (i.e.,  for  six 
tanks,  6x1)  is  permitted. 

c.  Col.  Ill  (To  be  added  by  subjects).  One  of  the  five 
change  conditions  shown  on  the  distributed  list  must 

be  entered  for  each  target  detected.  This  applies  only 
to  targets  located  within  the  common  ("overlap")  areas 
and  where  such  judgments  are  appropriate  (i.e.,  where 
has  previously  been  interpreted  and/or  reference 

comparison  material  is  provided  along  with  P^). 

d.  Col.  IV.  Indicate  the  confidence  you  have  that  the 
identification  of  the  detected  target  is  correct  by 
entering  one  of  the  letters  associated  with  the  five 
confidence  levels  for  each  target. 

e.  Col.  V.  Report  the  location  of  each  target  detected 
(whether  new,  gone,  unchanged,  etc.)  by  recording  the 
cell  number  (1-144)  corresponding  to  the  area  in  which 
the  target  is  located.  If  targets  subtend  more  than 
one  cell  or  intersect  a  grid  line,  report  both  grid 
coordinates.  Target  position  should  be  reported  as  it 
appears  on  the  more  recent  coverage  when  more  than  one 
coverage  is  provided. 

f.  Col.  VI.  Record  the  time  (to  the  nearest  minute)  when 
inspection  of  each  photo  or  photo-pair  was  begun  and 
the  time  when  search  was  terminated  (under  both  self- 
paced  or  experimenter  paced  conditions). 


2.  Annotated  Photographs.  With  colored  grease  pencil,  number¬ 
ing  each  target  consecutively;  indicate  the  precise  posi¬ 
tion  of  each  target  detected.  Annotate  only  the  more 
recent  coverage  when  more  than  one  photograph  is  provided. 
Indicate  the  location  of  targets  which  appear  only  on  the 
earlier  material  by  drawing  a  circle  around  that  location 
on  the  more  recent  coverage  in  which  the  target  would  have 
been  found  had  it  not  been  removed  during  the  intervening 
time  period.  Number  consecutively  all  annotations. 

5.  Situational  Overlays  (To  be  prepared  during  interpreta¬ 
tions  only).  Using  the  tracing  paper  provided,  prepare  an 
overlay  which  will  include  a  sufficient  amount  of  detail 
(terrain  features,  landmarks,  reference  points,  roads, 
rivers,  etc.)  to  insure  that  subsequent  users  can  readily 
localize  the  same  area  on  later  imagery  even  if  such  imagery 
is  of  different  scale  and  contains  only  partial  overlap. 
Include  all  targets  which  have  been  detected,  also  the 
identification  number  of  the  photograph  from  which  overlay 
was  prepared  and  the  name  of  the  subject. 

Additional  Points  of  Emphasis 

1.  BESRL- provided  PI  keys  could  be  used  as  needed  during 
testing. 

2.  "Change  Status"  responses  were  appropriate  only  for  targets 
falling  within  overlap  area,  although  all  targets  were  to 
be  reported  even  though  outside  common  area. 

3.  No  special  provisions  were  made  to  ensure  that  order  of 
photos  in  packets  was  constant  or  that  subjects  proceed 
systematically  through  the  10  photos  or  pairs. 

4.  Subjects  were  encouraged  to  provide  measures  of  individual 
performance  by  minimizing  communication  and  subject  inter¬ 
action  during  and  between  test  periods. 
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APPENDIX  C--CHANGE  STATUS  RESPONSE  ANALYSIS 


The  small  number  of  targets  representing  certain  types  of  change 
precludes  making  definitive  statements  about  targets  which  were  Unchanged, 

Moved,  or  Replaced.  For  instance,  there  were  only  three  "Unchanged" 
targets  in  the  image  sample,  and  45. 8  percent  of  the  times  they  were  pre¬ 
sented  they  were  not  detected.  "New"  targets,  of  which  there  were  85, 
were  correctly  recognized  as  such  two-thirds  of  the  time  and  missed  one- 
fourth  of  the  time.  "Gone"  targets,  on  the  other  hand,  although  more 
frequent  in  the  test  imagery,  were  correctly  recognized  less  than  one- 
half  the  time  and  omitted  one-half  the  time.  "Gone"  targets  are  appar¬ 
ently  much  more  difficult  under  the  OR  condition  for  no-experience  sub¬ 
jects  (see  Table  C-3).  This  circumstance  is  undoubtedly  attributable  to 
the  nature  of  the  reference  material  these  groups  are  required  to  use  or 
to  the  fact  that  the  110-experience  subjects  worked  with  overlays  and  re¬ 
ports  which  had  been  prepared  from  imagery  they  had  never  seen. 

"Gone"  targets  were  most  frequently  detected  and  categorized  cor¬ 
rectly  under  the  computer  compared  condition  (CC  reference  material).  j 

It  should  be  remembered,  however,  that  under  this  condition,  the  report  *  . 

prepared  on  the  P^  image  (which  contained  all  of  the  "Gone"  targets  as 

well  as  the  "Unchanged",  "Moved",  and  "Replaced"  targets)  was  accomplished 
without  a  time  limit.  The  effect  that  limited  search  time  has  on  the 
frequency  of  reporting  "Gone"  targets  can  readily  be  seen  by  comparing 
the  5-  versus  15-minute  conditions  for  the  other  reference  material  con¬ 
ditions.  Since  the  computer  was  programmed  to  compare  inputs  from  two 
independent  sources  ( and  P^)  and  make  change  status  assignments  based 

on  the  similarities  and  discrepancies  in  the  two  reports,  a  particularly 
complete  and  accurate  P  analysis  input  would  lead  to  a  preponderance  of 

correct  "Gone"  target  assignments,  whereas  more  complete  Pg  analysis 

would  produce  a  higher  incidence  of  correct  "New"  target  designations. 

When  "Unchanged"  targets  were  incorrectly  categorized,  they  were 
more  often  reported  as  "Gone"  targets  (46.9  percent)  than  as  "New" 
targets  (18.7  percent).  They  were  less  frequently  thought  to  be  "Moved" 
targets  (15*7  percent)  than  "Replaced"  targets  (18.7  percent).  The 
single  "Moved"  target  was  very  seldom  detected  and  when  it  was,  it  was 
usually  incorrectly  thought  to  be  a  "Gone"  target  indicating  difficulty 
in  detecting  and/or  recognizing  it  as  the  same  target  on  the  P^  imagery. 

Incorrect  change  status  responses  for  both  "New"  and  "Gone"  targets  were 
in  the  "Unchanged"  and  "Moved"  categories  (approximately  90  percent), 
suggesting  that  they  may  be  frequently  detected  on  either  P^  or  P^,  but. 

then  are  incorrectly  assumed  or  perceived  to  be  present  on  the  compara¬ 
tive  image  sample. 
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Table  C-l 


SUMMARY  DISTRIBUTION  FOR  CORRECT,  INCORRECT,  OMITTED, AND 
INVENTED  TARGET  RESPONSES  FOR  THE  FIVE 
CHANGE  STATUS  CATEGORIES 


-  —  ■ 

Change 

Status  Categories 

Target  Responses 

Unchanged 

Moved 

New 

Gone 

Repl aced 

( Percent ) 

(N  -  3)* 

(N  =  1) 

(N  =  85) 

(N  =  97) 

(N  =  0) 

Correct  Responses 

37-5 

3.1 

68.7 

43.5 

0.0 

Incorrect 

Responses 

16.7 

26.6 

4.2 

3-9 

0.0 

Omitted  (E  ) 

_  '  o 

Targets 

43.8 

70.3 

27.1 

52.6 

0.0 

Cumulative 

Totals 

100.0 

100.0 

100.0 

100.0 

0.0 

Inventive 

Errors  (E  ) 

(N  -  860) 

12.8 

3.1 

45.5 

38.3 

0.2 

aN  k  The  total  nuni bar  of  targets  on  ttte  ten  comparative  Photo  pairs  tailing  into  the  change  status  category. 


Table  C-2 

DISTRIBUTION  OF  TOTAL  CHANGE  STATUS  ERROR  RESPONSES 
ACROSS  CHANGE  STATUS  CONDITIONS 


Change  Status 
Responses  (Eca) 

True  Change  Status 

Unchanged 
(N  =  32)* 

Moved 

(N  =  17) 

New 

(N  =  .226) 

Gone 

(N  =  242) 

Replaced 
(N  •=  0) 

Unchanged 

11 .8 

25.7 

54.5 

0.0 

Moved 

15.7 

60.2 

37.2 

0.0 

New 

18.7 

5.9 

0.4 

0.0 

Gone 

46.9 

70.5 

5.5 

0.0 

Replaced 

18.7 

11.8 

8.8 

7-9 

*N  —  The  total  number  of  change  status  error  responses  |ECS)  for  each  change  status  category. 
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Asst.  Chief  of  Staff  for  Intelligence 
Office,  Chief  of  Research  and  Development 

The  present  publication  reports  on  a  study  of  methods  of  comparing  current  with 
prior  imagery  cover  of  a  given  area  as  means  of  detecting  change  in  the  status  of  tar¬ 
gets  in  the  area.  The  objective  of  the  research  effort,  undertaken  jointly  by  personnel 
of  the  Boeing  Company,  the  System  Development  Corporation,  and  of  the  INTERPRETER  TECH¬ 
NIQUES  Work  Unit,  USA  BESRL,  was  to  investigate  the  effects  of  selected  procedural  vari¬ 
ables  on  the  completeness  and  accuracy  with  which  change  in  the  status  of  targets  is 
detected.  Sixty- four  student  interpreters  from  USAINTS,  Fort  Holabird  participated  in 
performing  a  tactical  target  cnange  detection  task.  Sixteen  groups  (matched  on  scores 
on  the  GT  Aptitude  Area)  of  four  men  each  were  formed,  half  of  which  were  designated 
"experienced";  the  other  half,  "no-experlence".  Each  group  performed  the  target  search, 
target  identification,  and  change  detection  task  under  one  of  16  experimental  conditions! 
Comparison  materials  provided  were  either  pairs  of  unannotated  photon,  pairs  of  photos 
one  of  which  was  annotated,  or  photo  overlays  plus  written  reports  of  earlier  coverage 
paired  with  a  more  recent  photo.  In  a  fourth  procedure,  separate  analyses  of  the  com¬ 
parative  photos  were  made  and  the  data  subjected  to  a  simulated  computer  comparison. 

Each  comparative  scene  was  viewed  for  either  5  or  15  minutes.  Change  detection  per¬ 
formance  was  evaluated  in  terms  of  completeness  and  accuracy. 

Under  the  5_n>lnute  time  limit,  computerized  comparison  produced  more  complete 
change  detection  results  than  did  the  other  comparative  cover  analyses,  and  with  about 
the  same  degree  of  accuracy  as  comparison  by  an  interpreter.  The  advantage  of  computer 
comparison  was  less  marked  under  the  15-minute  time  limit.  Providing  the  interpreter 
with  overlay  plus  a  written  report  on  the  prior  imagery  was  the  least  effective  of  the 
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13.  ABSTRACT  -  Continued 

comparison  methods  investigated.  Without  the  actual  imagery  from  the 
earlier  mission,  the  interpreter  has  no  way  of  correcting  errors  in  the 
earlier  interpretation.  The  method  also  was  time-consuming.  Interpreter 
familiarity  with  the  area  through  preparation  of  prior  imagery  reports 
resulted  in  enhanced  performance  only  under  the  5-minute  limit.  Results 
thus  point  to  an  advantage  in  using  the  computerized  comparison  procedure 
in  an  advanced  surveillance  system.  For  maximum  operational  efficiency, 
interpreters  familiar  with  a  particular  area  should  be  assigned  to  make 
subsequent  comparative  cover  analysis  of  the  area. 
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